Exploration in the Dark: Reasoning about Planning Strategies

We present a new approach to reasoning about planning strategies in multiagent domains: an agent learns a boundedly-optimal planning strategy, relative to the current state of the environment and its goals, using a myopic attitude based on its repertoire of planning policies. In self-play, our approach also allows boundedly-optimal coordination. In terms of the exploration-exploitation tradeoff, the agent need not generate a model of its environment (nor of other domain agents) as part of its exploratory activities, which allows efficient exploration as well as tractability in real-world scenarios. As complement to the above approach, we present the Leap-and-Stride strategy, a novel probabilistic strategy that serves agents in situations where they can make no assumptions about their surroundings, either due to the lack of a priori knowledge about the environment, or because its complex and dynamic nature renders it inherently unpredictable. The Leap-and-Stride strategy interleaves exploratory activities into the agent’s planning logic, which enables it to converge to effective modes of interaction with its environment through trial-and-error. This strategy is tunable in the sense that the agent can control the intensity of its exploratory activities according to a risk management policy. We present theoretical results regarding the complexity of Leap-and-Stride, and demonstrate its viability through two sets of empirical studies.